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WWC Review of the Report “Interactive Learning Online at Public 
Universities: Evidence from a Six-Campus Randomized Trial”^ 

The findings from this review do not ref iect the fuii body of research evidence on interactive iearning oniine. 


What is this study about? 

The study investigated the effect of interactive learn- 
ing online (ILO), a form of online course instruction 
in which computer-guided instruction substitutes for 
some, though not all, traditional face-to-face instruc- 
tion. ILO-based courses are characterized by the use 
of student data to provide customized instruction 
to each student. Course instructors are also able to 
track students’ progress in detail in order to provide 
each student with individualized guidance relative to 
the goals of the course. The study was conducted 
at six public university campuses— two each from 
the State University of New York (SUNY); the City 
University of New York (CUNY) system; and the 
University of Maryland system. 

The study used data from 605 students enrolled 
in an introductory statistics courses at these insti- 
tutions. Fifty-two percent of the students were 
first-generation college students, 50% came from 
families earning less than $50,000 a year, and 31 % 
were racial/ethnic minorities. Of the 605 students 
in the study, 313 were randomly assigned to hybrid 
ILO course sections, in which computer-based 
instruction was supplemented by weekly 1 -hour face- 
to-face sessions where students could ask questions 
or be given targeted assistance by the instructor. The 
other 292 students were randomly assigned to sec- 
tions taught via traditional face-to-face instruction. 

Based on data provided by the participating universi- 
ties, the study authors examined the impact of ILO on 
the course pass rate (i.e., course completion with a 
passing grade).^ 


What did the study find? 

Pass rates in both the intervention and comparison 
course sections were similar (80% vs. 76%, respec- 
tively). The study authors reported, and the WWO 
confirmed, that there was no statistically significant 
difference between these rates. 


WWC Rating 


The research described in this 
report meets WWC group design 
standards without reservations 

This study is a well-executed randomized 
controlled trial. 

Although random assignment was utilized to form 
intervention and comparison groups at each of the 
six study sites, the specific recruitment protocol and 
incentive structure varied across sites. 


Features of Interactive Learning Online 


Interactive learning online uses computer-based 
instruction to allow students to learn course material 
independently. The main features are: 

• Interactive lessons, allowing students to track their 
own learning, 

• Embedded assessments in each instructional 
activity, and 

• Progress monitoring and targeted supplementary 
support by course instructor. 
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Appendix A: Study details 

Bowen, W. G., Chingos, M. M., Lack, K. A., & Nygren, T. I. (2013). Interactive learning online at public 
universities: Evidence from a six-campus randomized trial. Journal of Policy Analysis and 
Management, 33(1), 94-1 1 1 . 


Setting 

The study took place at six public universities in New York and Maryiand. 

Study sample 

The sampie inciuded 605 students. Most of the students were first-generation coilege stu- 
dents (52%), 50% were from families earning less than $50,000 a year, and a majority were 
female (58%). Approximately one-third (31 %) were racial/ethnic minorities. Fuli-time students 
comprised 90% of the sampie. 

Intervention 

group 

The intervention group in the study was comprised of students who were randomly assigned 
to sections of an introductory statistics course taught using interactive learning online (ILO). 
These sections were delivered in a “hybrid” mode, in which most of the instruction was 
delivered through interactive oniine materials, but the online instruction was supplemented 
by a weekly 1-hour face-to-face session, so that students could ask questions or be given 
targeted assistance by the instructor. 

Comparison 

group 

The comparison group in the study was comprised of students who were randomiy assigned 
to sections of the same introductory statistics course as the intervention group, but taught 
only using traditional face-to-face instruction. 

Outcomes and 
measurement 

The study reported findings for one eligible outcome— course pass rate. The outcome 
measure was based on student academic records obtained from the six universities 
participating in the study. 

Support for 
implementation 

The authors did not provide information on training received by ILO instructors or other 
support the institutions received for implementation. 

Reason for 
review 

This study was reviewed by the WWC in response to a request by the Institute of 
Education Sciences. 
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Appendix B: Outcome measure for the academic achievement domain 


Academic achievement 


Course pass rate This outcome is measured via a binary indicator of whether or not the student compieted the course and 

received a passing grade, it is based on academic record data obtained from the institutions participating 
in the study. 

study note: This study also assessed the impact of interactive learning online on three other outcomes — course completion, scores on a standardized test of statistics knowledge, 
and course final exam scores. None of these three outcomes were eligible for review under the Postsecondary Education review protocol, and therefore, are not included in this 
single study review. 
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Appendix C: Study findings for the academic achievement domain 


Mean 

(standard deviation) WWC calculations 


Domain and 
outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Academic achievement 

Course pass rate 

Full sample 

605 

students 

80% 

76% 

4% 

0,10 

+4 

>0.05 


Domain average for academic achievement 0.10 +4 Not 

statistically 

significant 


Table Notes: For mean difference, effecf size, and improvement index vaiues reported in the tabie, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on student outcomes, representing the average change expected for aii students 
who are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an aiternate presentation of the effect size, refiecting the 
change in an average student’s percentiie rank that can be expected if the student is given the intervention. The WWC-computed average effect size is a simpie average rounded 
to two decimai piaces; the average improvement index is caicuiated from the average effect size. The statisticai significance of the study's domain average was determined by 
the WWC; for exampie, the study is characterized as having a statisticaiiy significant positive effect because univariate statisticai tests are reported for each outcome measure, 
the effect for at ieast one measure within the domain is positive and statisticaiiy significant, and no effects are negative and statisticaiiy significant, accounting for muitipie 
comparisons. 

Study Notes: No corrections for ciustering or muitipie comparisons and no difference-in-differences adjustment were needed. The p-value presented here was reported in the 
originai study. Adjusted regression and standard error estimates were obtained from the study authors. These statistics form the basis of the effect size caicuiations. 


June 2014 


Page 4 



WWC Single Study Review 


Appendix D: Supplemental findings for the academic achievement domain 


Mean 

(standard deviation) WWC calculations 


Domain and 
outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Academic achievement 

Course pass rate 

Males 

256 

students 

79% 

75% 

4% 

0.09 

+4 

>0.05 

Course pass rate 

Females 

349 

students 

84% 

78% 

6% 

0.18 

+7 

>0.05 

Course pass rate 

First-generation 
college students 

315 

students 

77% 

75% 

2% 

0.05 

+2 

>0.05 

Course pass rate 

Not first- 

generation college 
students 

290 

students 

85% 

78% 

7% 

0.20 

+8 

>0.05 


Table Notes: The supplemental findings presented in this tabie are additionai findings that do not factor into the determination of the evidence rating. For mean difference, effecf 
size, and improvement index vaiues reported in the tabie, a positive number favors the intervention group and a negative number favors the comparison group. The effect size is a 
standardized measure of fhe effect of an intervenfion on studenf outcomes, representing the average change expected for aii sfudenfs who are given the intervention (measured in 
standard deviations of the outcome measure). The improvement index is an aiternate presentation of the effect size, rejecting the change in an average student's percentiie rank 
that can be expected if the student is given the intervention. 

Study Notes: No corrections for ciustering or muitipie comparisons and no difference-in-differences adjusfmenf were needed. The p-vaiues presented here were reported in the 
originai study. Using data obtained from the study authors, tests for subgroup interactions reveaied no statisticaiiy significant differences in effects across subgroups. 
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Endnotes 

■' Single study reviews examine evidence published in a study (supplemented, if necessary, by information obtained directly from the 
authors) to assess whether the study design meets WWC evidence standards. The review reports the WWC’s assessment of whether 
the study meets WWC evidence standards and summarizes the study findings following WWC conventions for reporting evidence on 
effectiveness. This study was reviewed using the Postsecondary Education topic area review protocol, version 2.0. The WWC rating 
applies only to the results that were eligible under this topic area and met WWC standards, and not necessarily to all results presented 
in the study. 

^ There were three outcomes included in the study that are not described in this WWC report— course completion, scores on a stan- 
dardized test of statistics, and course final exam scores. See the study note in Appendix B for more information. 

Recommended Citation 

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2014, June). 

WWC review of the report: Interactive learning online at public universities: Evidence from a six-campus 
randomized trial. Retrieved from http://whatworks.ed.gov 
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Glossary of Terms 

Attrition 


Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Improvement index 


Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Single-case design 
(SCD) 

Standard deviation 


Statistical significance 
Substantively important 


Attrition occurs when an outcome variable is not avaiiabie for aii participants initiaiiy assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review if it falls within the scope of the review protocol and uses either 
an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

Along a percentile distribution of students, the improvement index represents the gain 
or loss of the average student due to the intervention. As the average student starts at 
the 50th percentile, the measure ranges from -50 to +50. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which subjects are assigned 
to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which investigators randomly assign 
eligible participants into intervention and comparison groups. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 

The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample are spread out over a large range of values. 

Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% (p < 0.05). 

A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 


Please see the WWC Procedures and Standards Handbook (version 3.0) for additional details. 
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